Efficient Reinforcement Learning in Continuous Environments

Reinforcement Learning (RL) is a machine learning paradigm with which autonomous agents can improve their behavior in unknown environments based on their own experience without an explicit teacher signal. RL algorithms are based on estimating a value function over the state space, and scaling them to large state spaces remains a challenge. One approach, known as variable resolution, is to focus representational power in regions of the state space where experience shows it is most needed. One of the most promising variable resolution methods is partigame, which has competitive performance and has shown promising scalability on the class of deterministic, continuous, goal-type RL problems. It performs dynamic, kd-tree-based partitioning of the state space based on a game-theoretic approach to assigning costs to partitions, and it uses an a priori local controller to try to navigate through the partitions it creates to reach the goal. Despite its promise, however, parti-game has a number of shortcomings relating to efficiency, consistency, sub-optimality of solutions found and reliance on a designer-supplied local controller. This work introduces a family of variable resolution algorithms, in the same spirit of parti-game, that addresses each of these drawbacks, thus providing a powerful, a-priori -model-independent paradigm for finding higher quality solutions of